Output csv #61
Merged
Output csv #61
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
forsyth2
commented
Jul 15, 2025
Collaborator
Author
forsyth2
left a comment
forsyth2
left a comment
There was a problem hiding this comment.
@TonyB9000 You only need to review the final commit (baacf8d)
| @@ -0,0 +1,2 @@ | |||
| This directory contains the csv files that have output matching the tables on the e3sm_data_docs webpage. | |||
Collaborator
Author
There was a problem hiding this comment.
The csv files are placed in this directory. That means they will be available at https://github.com/E3SM-Project/e3sm_data_docs/blob/main/machine_readable_data/. If you approve of the PR's output, I can produce the csv files for v2 and v2.1 too, if needed.
| @@ -0,0 +1,91 @@ | |||
| Simulation,Data Size (TB),ESGF Links,HPSS Path | |||
| 20180129.DECKv1b_piControl.ne30_oEC.edison,40,https://esgf-node.ornl.gov/search?project=CMIP6&activeFacets=%7B%22institution_id%22%3A%22E3SM-Project%22%2C%22source_id%22%3A%22E3SM-1-0%22%2C%22experiment_id%22%3A%22piControl%22%2C%22variant_label%22%3A%22r1i1p1f1%22%7D,/home/projects/e3sm/www/WaterCycle/E3SMv1/LR/20180129.DECKv1b_piControl.ne30_oEC.edison | |||
Collaborator
Author
There was a problem hiding this comment.
2 important notes:
- The ESGF link is stripped of human readable text -- it's just the link.
- The HPSS path never has "(symlink) " in it, as the web page has.
Comment on lines
+353
to
356
| construct_output_csv(resolutions, header_cells, f"../machine_readable_data/{model_version}_{group_name}_simulations.csv") | ||
| print(f"csv of the simulations will be available at https://github.com/E3SM-Project/e3sm_data_docs/blob/main/machine_readable_data/{model_version}_{group_name}_simulations.csv") | ||
| # TODO: add proper subdirs and index.rst files in docs/ | ||
| generate_table( |
Collaborator
Author
There was a problem hiding this comment.
Really, we're just creating two output types here (1. a csv for a computer to read and 2. a rst and ultimately html page for a human to read) from the same data in RAM. This implementation doesn't construct the HTML from the csv directly.
Collaborator
There was a problem hiding this comment.
@forsyth2 Understood. This will be great. (If the very same process is creating both, there should be little room for inconsistenies.)
The only fields I need are:
- A "unique key" to identify the archive (essentially the unique case_id)
- The Volume-in-TB field (to avoid download of M TB, when disk free space < M TB)
- The precise HPSS path (to feed "zstash --check --hpss=path) when zstash is ready to perform this transfer.
I made a wild guess for v3: It looks like this:
LR:DECK,v3.LR.piControl,1,na,na,/home/g/golaz/E3SMv3.LR/v3.LR.piControl
LR:DECK,v3.LR.1pctCO2_bcdt15m,1,na,na,/home/w/wlin/E3SMv3/v3.LR.1pctCO2_bcdt15m
LR:DECK,v3.LR.abrupt-4xCO2_bcdt15m,1,na,na,/home/w/wlin/E3SMv3/v3.LR.abrupt-4xCO2_bcdt15m
LR:Historical,v3.LR.historical_0051,1,na,na,/home/w/wlin/E3SMv3/v3.LR.historical_0051
LR:Historical,v3.LR.historical_0101,1,na,na,/home/w/wlin/E3SMv3/v3.LR.historical_0101
LR:Historical,v3.LR.historical_0151,1,na,na,/home/w/wlin/E3SMv3/v3.LR.historical_0151
LR:Historical,v3.LR.historical_0201,1,na,na,/home/w/wlin/E3SMv3/v3.LR.historical_0201
LR:Historical,v3.LR.historical_0251,1,na,na,/home/w/wlin/E3SMv3/v3.LR.historical_0251
LR:AMIP,v3.LR.amip_0101,1,na,na,home/w/wlin/E3SMv3/AMIP/v3.LR.amip_0101
LR:AMIP,v3.LR.amip_0151,1,na,na,home/w/wlin/E3SMv3/AMIP/v3.LR.amip_0151
LR:AMIP,v3.LR.amip_0201,1,na,na,home/w/wlin/E3SMv3/AMIP/v3.LR.amip_0201
LR:DAMIP,v3.LR.hist-GHG_0101,1,na,na,/home/g/golaz/E3SMv3.LR/v3.LR.hist-GHG_0101
LR:DAMIP,v3.LR.hist-GHG_0151,1,na,na,/home/g/golaz/E3SMv3.LR/v3.LR.hist-GHG_0151
LR:DAMIP,v3.LR.hist-GHG_0201,1,na,na,/home/g/golaz/E3SMv3.LR/v3.LR.hist-GHG_0201
LR:DAMIP,v3.LR.hist-aer_0101,1,na,na,/home/g/golaz/E3SMv3.LR/v3.LR.hist-aer_0101
LR:DAMIP,v3.LR.hist-aer_0151,1,na,na,/home/g/golaz/E3SMv3.LR/v3.LR.hist-aer_0151
LR:DAMIP,v3.LR.hist-aer_0201,1,na,na,/home/g/golaz/E3SMv3.LR/v3.LR.hist-aer_0201
LR:DAMIP,v3.LR.hist-xGHG-xaer_0101,1,na,na,/home/g/golaz/E3SMv3.LR/v3.LR.hist-xGHG-xaer_0101
LR:DAMIP,v3.LR.hist-xGHG-xaer_0151,1,na,na,/home/g/golaz/E3SMv3.LR/v3.LR.hist-xGHG-xaer_0151
LR:DAMIP,v3.LR.hist-xGHG-xaer_0201,1,na,na,/home/g/golaz/E3SMv3.LR/v3.LR.hist-xGHG-xaer_0201
LR:RFMIP,v3.LR.piClim-control-iceini,1,na,na,/home/k/kaizhang/E3SM/E3SMv3/v3.LR.piClim-histall/v3.LR.piClim-control-iceini
LR:RFMIP,v3.LR.piClim-histall_0101,1,na,na,/home/k/kaizhang/E3SM/E3SMv3/v3.LR.piClim-histall/v3.LR.piClim-histall_0101
LR:RFMIP,v3.LR.piClim-histall_0151,1,na,na,/home/k/kaizhang/E3SM/E3SMv3/v3.LR.piClim-histall/v3.LR.piClim-histall_0151
LR:RFMIP,v3.LR.piClim-histall_0201,1,na,na,/home/k/kaizhang/E3SM/E3SMv3/v3.LR.piClim-histall/v3.LR.piClim-histall_0201
LR:RFMIP,v3.LR.piClim-histGHG_0101,1,na,na,/home/k/kaizhang/E3SM/E3SMv3/v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0101
LR:RFMIP,v3.LR.piClim-histGHG_0151,1,na,na,/home/k/kaizhang/E3SM/E3SMv3/v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0151
LR:RFMIP,v3.LR.piClim-histGHG_0201,1,na,na,/home/k/kaizhang/E3SM/E3SMv3/v3.LR.piClim-histGHG/v3.LR.piClim-histGHG_0201
LR:RFMIP,v3.LR.piClim-histaer_0101,1,na,na,/home/k/kaizhang/E3SM/E3SMv3/v3.LR.piClim-histaer/v3.LR.piClim-histaer_0101
LR:RFMIP,v3.LR.piClim-histaer_0151,1,na,na,/home/k/kaizhang/E3SM/E3SMv3/v3.LR.piClim-histaer/v3.LR.piClim-histaer_0151
LR:RFMIP,v3.LR.piClim-histaer_0201,1,na,na,/home/k/kaizhang/E3SM/E3SMv3/v3.LR.piClim-histaer/v3.LR.piClim-histaer_0201
TonyB9000
approved these changes
Jul 15, 2025
Collaborator
TonyB9000
left a comment
TonyB9000
left a comment
There was a problem hiding this comment.
This looks like a LOT of work. It looks good to me, although my judgement is impaired ... Proof on the pudding, as they say.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Follow-up to #60 to address
First 5 commits are from #60 and will disappear once that's merged. baacf8d is the important commit.